The present invention relates generally to verification of complex workflows and, more particularly, to evaluating the quality of a complex workflow, for example those arising in research and development, through the subdivision of the complex workflow into verifiable modules, whose verification is done by internal assessment or by leveraging a community-based assessment.
A complex workflow consists of a number of inputs in the form of (but not limited to) data, signals or material, for example, and a set of processing steps which yield a number of desired outputs in the form of signals, materials or data. These outputs depend both on the inputs and on the processing steps in complex and nontrivial ways. Hence, the goodness or appropriateness of the overall workflow design cannot be assessed using simple or trivial metrics based on the final output. This is so because if the final output is not what is desired, it would be very difficult to understand which step or steps in the complex workflow are at the root of the failure to produce the expected outputs.
In the present context, verification generally refers to the process of evaluating a system, component or other product or research output at the end of its development cycle to determine whether it meets projected performance or accuracy goals (http://medical-dictionary.thefreedictionary.com/verification). Another definition of verification in the context of research is what is usually considered as positive or negative controls. Here, verification is the determination that a module accurately reproduces well established (positive or negative) results.
Formal verification may be helpful in proving the correctness of systems such as, for example, cryptographic protocols, combinational circuits, digital circuits with internal memory, and software expressed as source code (see, e.g., http://en.wikipedia.org/wiki/Formal_verification). The verification of these systems is done by providing a formal proof on an abstract mathematical model of the system, with the correspondence between the mathematical model and the nature of the system being otherwise known by construction. Examples of mathematical objects often used to model systems include finite state machines, labeled transition systems, Petri nets, timed automata, hybrid automata, process algebra, formal semantics of programming languages such as operational semantics, denotational semantics, axiomatic semantics and Hoare logic.
While formal (mathematical or logical) verification may be helpful in proving the correctness of systems with limited scope (e.g., finite state machine, source code, cryptography, etc.), the general problem of verifying the correctness of a complex research plan is more ad hoc with few established techniques. The goodness or fitness of a complex workflow cannot be estimated using simple or trivial metrics based on the overall process. Indeed, verification can be difficult and time consuming, and even in systems with limited scope, a module or component may not be verified for absolute correctness unless that particular module or component is highly critical for system reliability or security. Therefore, methods that provide for a more approximate verification will be needed for complex systems. Approximate verification methods often include a reference or stereotyped input that tests only a limited range of the input state space. The output is tested against a known or idealized output.